Operators in the oil and gas sector often grapple with significant hurdles when trying to access subsurface data archived on tape media. Much of this information was stored over 25 years ago on various tape formats, including 9-track tapes and LTO cartridges. These extensive collections remain locked away in offsite tape vaults, missing out on the transformative advantages of cloud computing.
Accessing subsurface data is cumbersome and slow when it is stored on tapes. Additionally, the ongoing costs associated with maintaining data in tape vaults can be exorbitantly high compared to modern cloud storage solutions. The data trapped on outdated tapes hampers innovation and agility, and limits the scalability options available through Amazon Web Services (AWS). Often regarded as “dark” data, these collections are not integrated into business workflows, rendering them invisible to users and unable to leverage contemporary scientific algorithms or the expansive potential of cloud computing.
Tape Ark, with its roots in the oil and gas sector, is well-acquainted with the challenges of liberating various tape formats dating back to the 1960s and 70s. The company specializes in facilitating the secure migration of this data to AWS, providing essential services for oil and gas companies. Customers frequently possess immense libraries of tapes, sometimes numbering tens of thousands or even millions, containing data volumes that reach into the tens of petabytes. Conducting tape migration on such a scale is an intricate and resource-heavy process, yet it is crucial for firms seeking to unlock the full value of their data. Tape Ark, as an AWS Select Technology Partner, is uniquely positioned to manage large-scale migrations efficiently.
Migrating Data from Tapes to the Cloud
Before the advent of AWS cloud services, oil and gas companies depended on tapes to transfer data to and from boats, platforms, and exploration vessels. Given the long retention requirements in the industry, these tapes are typically archived in long-term storage facilities until needed again. The industry standard necessitates moving tape-bound data to more efficient storage every 3-5 years to maintain accessibility. Since tapes are not random access storage mediums, the organization of data on tapes is critical for efficient retrieval.
Currently, much of the legacy seismic data in the energy sector remains underutilized, primarily due to its storage on tape media. By transitioning this data to the cloud, oil and gas companies can derive greater value, enabling the application of artificial intelligence (AI) and machine learning (ML) techniques for seismic analysis and advanced processing. Tape Ark’s capabilities have led to successful tape-to-cloud migrations for numerous oil and gas operators. For instance, a recent project involved processing tasks on AWS that utilized 1 million virtual CPUs (vCPUs), achieving unprecedented results that could not be matched with tape-stored data. Ongoing initiatives with Tape Ark are focused on migrating tens of millions of tapes and over 400 petabytes of exploration data.
Solution Overview
Tape Ark and AWS have crafted a solution designed to liberate subsurface data, making it accessible for analytics, machine learning, and collaborative workflows. The workflow begins with receiving tape media and conducting a thorough audit, allowing companies to gauge their prospective cloud footprint accurately, identify duplicates, and exclude data tied to joint ventures or third parties.
Following the audit, all data is ingested into the designated cloud account using Tape Ark’s advanced technology stack. During the ingestion process, automated checksum and name validation checks are performed to ensure quality control. Clients can prepare these checks in advance, facilitating real-time quality assessments. After ingestion, data tiering policies automatically transfer data to the requested tier, with JSON metadata manifest files generated and stored in client AWS accounts to update internal databases. Completed tapes can then be disposed of through Tape Ark’s certified disposal services or returned to the client.
The internal rapid mass tape ingestion platform, Arkbridge, was developed to leverage AWS Cloud’s security and strength. This system provides an automated Internet of Things (IoT) approach that eliminates manual processes for ingesting media into AWS client accounts, ensuring high accuracy and efficiency. By harnessing Amazon API Gateway and AWS Lambda functions, the metadata of ingested tapes is stored in Amazon Relational Database Service (Amazon RDS).
The solution incorporates a unique blend of image processing using Amazon Textract and AWS IoT Core to manage tape reads, streamlining the ingestion process at scale. Subsurface tape data is converted into objects stored in Amazon Simple Storage Service (Amazon S3), either through direct tape-to-cloud ingestion or via AWS Snowball. The data objects are stored in Amazon S3 Standard for rapid access and Amazon S3 Glacier for deep archiving. Ultimately, this object data in S3 serves as the foundation for clients’ seismic and machine learning applications.
Conclusion
This article illustrates Tape Ark’s effective strategy in generating value from subsurface data stored on legacy tapes by ingesting and cataloging it to AWS. With subsurface data transitioned to the cloud, oil and gas operators can enhance productivity and gain timely insights from previously inaccessible “dark” data. This enables them to reanalyze their data and apply AI/ML workflows, reintegrating their dark data into modern operational processes. For further insights, check out this another blog post for more details.
To learn more about how Tape Ark can help you preserve and migrate your tape-bound data to the cloud and maximize its value, visit this valuable authority on this topic. Additionally, for those interested in gaining knowledge about fulfillment centers and training new hires, this is an excellent resource.
Leave a Reply